The Indifferent Naive Bayes Classifier

The Naive Bayes classifier is a simple and accurate classifier. This paper shows that assuming the Naive Bayes classifier model and applying Bayesian model averaging and the principle of indifference, an equally simple, more accurate and theoretically well founded classifier can be obtained. Introduction In this paper we use Bayesian model averaging and the principle of indifference to derive an improved classifier which we name Indifferent Naive Bayes classifier (IndifferentNB from now on). First we introduce the Naive Bayes model, paying special attention to its conditional independence assumptions and to the estimation of its parameters. Second, we introduce Naive distributions and show that they are conjugate with respect to the Naive Bayes model and that they can be integrated in closed form to get averaged predictions. Third, we apply the principle of indifference, getting the final expression for IndifferentNB. Fourth, we perform an empirical comparison of IndifferentNB with the standard implementation of Naive Bayes and the one proposed in (Kontkanen et al. 1998) showing that IndifferentNB reduces the classification error rate and approximates the probabilities better, specially when few data is available. We finish with some conclusions and possibilities for future research. The Naive Bayes model The Naive Bayes classifier (Langley, Iba, & Thompson 1992) is a classification method based on the assumption of conditional independence between the different variables in the dataset given the class. Following the notation in (Cowell et al. 1999), being X , Y and Z random variables we will write X ⊥⊥ Y|Z for “X is conditionally independent on Y given Z”. In this notation, the Naive Bayes model states that ∀i, j 1 ≤ i, j ≤ n ; Ai ⊥⊥ Aj |C (1) The Naive Bayes model as a Bayesian network As can be seen in (Cowell et al. 1999) and in (Friedman, Geiger, & Goldszmidt 1997) in terms of Bayesian networks, Copyright c © 2003, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. the Naive Bayes model can be represented as the network in Figure 1. The Bayesian network in Figure 1 is not the